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Abstract 

We survey the diverse approaches to the notion of information content: 
from Shannon entropy to Kolmogorov complexity. The main applications 
of Kolmogorov complexity are presented: namely, the mathematical no- 
tion of randomness (which goes back to the 60's with the work of Martin- 
Lof, Schnorr, Chaitin, Levin), and classification, which is a recent idea 
with provocative implementation by Vitanyi and Cilibrasi. . 

Note. Following Robert Scare's recommendations in [35| , which have now gained 
large agreement, we shall write computable and computably enumerable in place 
of the old fashioned recursive and recursively enumerable. 

Notation. By log(a;) we mean the logarithm of x in base 2. By [_x\ we mean the 
"floor" of X, i.e. the largest integer < x. Similarly, \x~\ denotes the "ceil" of x, 
i.e. the smallest integer > x. Recall that the length of the binary representation 
of a non negative integer n is 1 -I- [log nj . 



1 Three approaches to the quantitative defini- 
tion of information 

A title borrowed from Kolmogorov's seminal paper, 1965 [55]. 

1.1 Which information ? 
1.1.1 About anything... 

About anything can be seen as conveying information. As usual in mathematical 
modelization, we retain only a few features of some real entity or process, and 
associate to them some finite or infinite mathematical objects. For instance. 
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• - an integer or a rational number or a word in some alphabet, 

- a finite sequence or a finite set of such objects, 

- a finite graph,... 

• - a real, 

- a finite or infinite sequence of reals or a set of reals, 

- a function over words or numbers,... 

This is very much as with probability spaces. For instance, to modelize the 
distributions of 6 balls into 3 cells, (cf. Feller's book [T3] §12, 115) we forget 
everything about the nature of balls and cells and of the distribution process, 
retaining only two questions: "how many are they?" and "are they distinguish- 
able or not?" . Accordingly, the modelization considers 

- either the 729 = 3® maps from the set of balls into the set of cells in case the 
balls are distinguishable and so are the cells (this is what is done in Maxwell- 
Boltzman statistics), 

- or the 28 = triples of non negative integers with sum 6 in case the cells 
are distinguishable but not the balls (this is what is done in in Bose-Einstein 
statistics) 

- or the 7 sets of at most 3 integers with sum 6 in case the balls are undistin- 
guishable and so are the cells. 

1.1.2 Especially words 

In information theory, special emphasis is made on information conveyed by 
words on finite alphabets. I.e. on sequential information as opposed to the 
obviously massively parallel and interactive distribution of information in real 
entities and processes. A drastic reduction which allows for mathematical de- 
velopments (but also illustrates the Italian sentence "traduttore, traditore!"). 

As is largely popularized by computer science, any finite alphabet with more 
than two letters can be reduced to one with exactly two letters. For instance, 
as exemplified by the ASCII code (American Standard Code for Information 
Interchange), any symbol used in written English - namely the lowercase and 
uppercase letters, the decimal digits, the diverse punctuation marks, the space, 
apostrophe, quote, left and right parentheses - can be coded by length 7 binary 
words (corresponding to the 128 ASCII codes). Which leads to a simple way to 
code any English text by a binary word (which is 7 times longer) 

Though quite rough, the length of a word is the basic measure of its information 
content. Now a fairness issue faces us: richer the alphabet, shorter the word. 
Considering groups of k successive letters as new letters of a super-alphabet, one 
trivially divides the length by k. For instance, a length n binary word becomes 
a length [jfel "^ord with the usual packing of bits by groups of 8 (called bytes) 
which is done in computers. 

This is why length considerations will always be developed relative to binary 

^For other European languages which have a lot of diacritic marks, one has to consider the 
256 codes of the Extended ASCII code. 
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alphabets. A choice to be considered as a normalization of length. 

Finahy, we come to the basic idea to measure the information content of a math- 
ematical object X : 



What do we mean precisely by "encodes" is the crucial question. Following the 
trichotomy pointed by Kolmogorov [22] . we survey three approaches. 




1.2 Combinatorial approach 

1.2.1 Constant-length codes 

Let's consider the family A" of length n words in an alphabet A with s letters 
oi, a^. Coding the Oi's by binary words Wi's all of length [log s~\ , to any word 
u in A" we can associate the binary word ^ obtained by substituting the Wi's 
to the occurrences of the a^'s in u. Clearly, ^ has length n[logs]. Also, the 
map M I— > f is very simple. Mathematically, it can be considered as a morphism 
from words in alphabet A to binary words relative to the algebraic structure (of 
monoid) given by the concatenation product of words. 

Observing that n log s can be smaller than n [log s] , a modest improvement is 
possible which saves about n[logs] — nlogs bits. The improved map u i— > ^ 
is essentially a change of base: looking at u as the base s representation of an 
integer fc, the word ^ is the base 2 representation of k. Now, the map w i-^ ^ is 
no more a morphism. However, it is still quite simple and can be computed by 
a finite automaton. 

We have to consider fc-adic representations rather than the usual fc-ary 
ones. The difference is simple: instead of using digits 0, 1, k — 1 use 
digits 1, fc. The interpretation as a sum of successive exponentials of k 
is unchanged and so are all usual algorithms for arithmetical operations. 
Also, the lexicographic ordering on fc-adic representations corresponds 
to the natural order on integers. For instance, the successive integers 

0. 1, 2, 3, 4, 5, 6, 7, written 0, 1, 10, 11, 100, 101, 110, 111 in binary (i.e. 2- 
ary) have 2-adic representations the empty word (for 0) and then the words 

1, 2, 11, 12, 21, 22, 111. Whereas the length of the fc-ary representation 
of a; is 1 -I- [jfjIJ i its fc-adic representation has length [ '"^^^"j^^-* J . 

Let's interpret the length n word u as the s-adic representation of an 
integer x between t = s"~^ -I- ... -f -f- s -I- 1 and t' ^ s" + ... + s'^ + s 
(which correspond to the length n words 11...1 and ss...s). Let ^ be the 
2-adic representation of this integer x. The length of ^ is < [log(f' -I- 1)J = 

Llog(^^^)J < L(n + l)logs-log(s-l)J = L'ilogs-log(l-i)J which 
differs from nlogs by at most 1. 

1.2.2 Variable-length prefix codes 

Instead of coding the s letters of A by binary words of length [log s] , one can 
code the Ci's by binary words Wi's having different lengthes so as to associate 
short codes to most frequent letters and long codes to rare ones. Which is the 



information content of x 



length of a shortest binary word 
which "encodes" x 
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basic idea of compression. Using these codes, the substitution of the Wi's to 
the occurrences of the a^'s in a word u gives a binary word ^. And the map 
zi I— > ^ is again very simple. It is still a morphism from the monoid of words on 
alphabet A to the monoid of binary words and can also be computed by a finite 
automaton. 

Now, we face a problem: can we recover u from ^ ? i.e. is the map u i— > ^ 
injective? In general the answer is no. However, a simple sufficient condition 
to ensure decoding is that the family wi,...,Ws be a so-called prefix- free code. 
Which means that if i ^ j then Wi is not a prefix of Wj . 

This condition insures that there is a unique Wi^ which is a prefix of ^. 
Then, considering the associated suffix of v (i.e. v — Wi^^i) there is a 
unique which is a prefix of ^i, i.e. u is of the form u — 'Wi^Wi2(^2- And 
so on. 

Suppose the numbers of occurrences in u of the letters ai, Os are toi, TOs, 
so that the length of m is n = mi + ... + rris- Using a prefix-free code wi, ...,Ws, 
the binary word ^ associated to u has length toi|k;i| -|- ... + ms\ws\. A natural 
question is, given mi, ...,ms, how to choose the prefix-free code wi, ...^Ws so as 
to minimize the length of ^ ? 

David A. Huffman, 1952 found a very efficient algorithm (which has linear 
time complexity if the frequencies are already ordered) . This algorithm (suitably 
modified to keep its top efficiency for words containing long runs of the same 
data) is nowadays used in nearly every application that involves the compression 
and transmission of data: fax machines, modems, networks,... 

1.2.3 Entropy of a of distribution of frequencies 

The intuition of the notion of entropy in information theory is as follows. 
Given natural integers toi, ...,ms, consider the family J-mi,....ms of length n = 
mi -!-... + nis words of the alphabet A in which there are exactly mi, ...,ms 
occurrences of letters ai, a^.. How many binary digits are there in the binary 
representation of the number of words in J-mi....,ms ? It happens (cf. Proposi- 
tion [2]) that this number is essentially linear in n, the coefficient of n depending 
solely on the frequencies It is this coefficient which is called the 

entropy H of the distribution of the frequencies 

Now, H has a striking significance in terms of information content and com- 
pression. Any word u in !Fmi,...,ms is uniquely characterized by its rank in this 
family (say relatively to the lexicographic ordering on words in alphabet A) . In 
particular, the binary representation of this rank "encodes" u and its length, 
which is bounded by nH (up to an 0(log n) term) can be seen as an upper bound 
of the information content of u. Otherwise said, the n letters of u are encoded 
by nH binary digits. In terms of compression (nowadays so popularized by the 
zip-like softwares), u can be compressed to nH bits i.e. the mean information 
content (which can be seen as the compression size in bits) of a letter of u is H. 

Definition 1 (Shannon, 1948 [34]). Let /i, /s be a distribution of frequencies, 
i.e. a sequence of reals in [0, 1] such that /i + ... + /« = 1. The entropy of fi, fs 
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is the real 

if--(/ll0g(/l) + ... + /sl0g(/.)) 

Let's look at two extreme cases. 
If all frequencies are equal to ^ then the entropy is log(s), so that the mean 
information content of a letter of u is log(s), i.e. there is no better (prefix-free) 
coding than that described in m.2.11 

In case one frequency is 1 and the other ones are 0, the information content of u is 
reduced to its length n, which, written in binary, requires log(n) bits. As for the 
entropy, it is (with the usual convention OlogO = 0, justified by the fact that 
linix^o X log a; = 0) . The discrepancy between nH = and the true information 
content logn comes from the O(logn) term (cf. the next Proposition). 

Proposition 2. Let mi, ...,ms be natural integers and n = mi + ... + 111^. Then, 
letting H be the entropy of the distribution of frequencies the number 

tt-^mi,...,rris of words in m. Satisfies 

log(tJJ^mi,...,mJ ^nH + O(logn) 

where the bound in O(logn) depends solely on s and not on mi, ...,ms. 

Proof. J-mi,...,ms contains rni'.x"' xm '. words. Using Stirling's approxi- 
mation of the factorial function (cf. Feller's book [16)). namely x\ = 
\/2tv x^~^^ e~"'"'"T2 where Q < 9 < 1 and equality n = mi -I- ... -I- ms, we 
get 

log( ^^l x"' X m,! ^ " (^mOlog(n)-(^maogmO 

1 i 

+ llog{ )- (s - 1) log \/2^ + a 

2 mi X ... X nis 

where |a| < lege. The first two terms are exactly n[^^ ^ log(^)] = 
nH and the remaining sum is O(logn) since n^~'' < ^ " xm — ^' '-' 

1.2.4 Shannon's source coding theorem for symbol codes 

The significance of the entropy explained above has been given a remarkable 
and precise form by Claude Elwood Shannon (I9I6-200I) in his celebrated 1948 
paper [34]. It's about the length of the binary word ^ associated to u via a 
prefix-free code. Shannon proved 

- a lower bound of |^| valid whatever be the prefix- free code wi, ...,Ws, 

- an upper bound, quite close to the lower bound, valid for particular prefix-free 
codes wi, (those making ^ shortest possible, for instance those given by 
Huffman's algorithm). 

Theorem 3 (Shannon, 1948 [34]). Suppose the numbers of occurrences in u of 
the letters ai, as are mi, ...,ms. Let n — mi + ... + m^. 

1. For every prefix- free sequence of binary words Wi, ...,Ws, the binary word ^ 
obtained by .substituting Wi to each occurrence of a,; in u satisfies 

nH < \C\ 
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where H = -(^ log(^) + ... + ^ log(^)) is the so-called entropy of the 
considered distribution of frequencies 

2. There exists a prefix-free sequence of binary words Wi, such that 

nH < \^\ < n{H+l) 
Proof. First, we recall two classical results. 

Theorem (Kraft's inequality). Let £i, £3 be a finite sequence of inte- 
gers. Inequality 2~^^ + ... + 2~^= < 1 holds if and only if there exists a 
prefix-free sequence of binary words wi, ...,Ws such that £1 = \wi\, £s = 

\Wa\. 

Theorem (Gibbs' inequality). Let pi, ...,ps and q\, ...,qs be two proba- 
bility distributions, i.e. the pi's (resp. qi's) are in [0,1] and have sum 1. 
Then ^ '^Pilog{pi) < — '^Pilog{qi) with equality if and only if pi = qi 
for all i. 

Proof of L Set Pi = ^ and qi = where 5" = 2-l"''l. Then 

Id = E,'">.I =^[E.^(-iog(gO-iog5)] 

> '^[-(E, ^ log(^) - log S] = n[H - log S] > nH 

The first inequality is an instance of Gibbs' inequality. For the last one, 
observe that S < 1 and apply Kraft' inequality. 

Proof of 2. Set = [-log(^)]. Observe that 2"^- < ^. Thus, 
2~^^ + ... + 2^^' < 1. Applying Kraft inequality, we see that there exists 
a prefix- free family of words wi, ...,Ws with lengthes £1, ...,£s. 
We consider the binary word ^ obtained via this prefix-free code, i.e. ^ 
is obtained by substituting Wi to each occurrence of Oi in u. Observe 
that -log(^) < £i < -log(^) + 1. Summing, we get nH < 1^ < 
n{H-\-l). " " p 

In particular cases, the lower bound nH is exactly |^|. 

Theorem 4. In case the frequencies ^ 's are all negative powers of two (i.e. 
|, i, then the optimal ^ (given by Huffman algorithm) satisfies £, — nH. 

1.2.5 Closer to the entropy 

As simple as they are, prefix-free codes are not the only way to efficiently encode 
into a binary word ^ a word u from alphabet ai, ...,as for which the numbers 
TTii, rus (of occurrences of the a^'s) are known. Let's go back to the encoding 
mentioned at the start of i jl.2.31 A word u in the family J^mi,...,ms (of length 
n words with exactly mi, ...,ms occurrences of oi, Cg) can be recovered from 
the following data: 

- the values of mi, ...,ms, 

- the rank of u in J-mi (relative to the lexicographic order on words). 

We have seen (cf. Proposition (2]) that the rank of u has a binary representation 
p of length < nH + O(logri). The integers mi,...,ms are encoded by their 
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binary representations fii,...,fis which are all < 1 + [lognj. Now, to encode 
mi,...,ms and the rank of u, we cannot just concatenate /Xg, p : how 

would we know where /ii stops, where ^2 starts,..., in the word obtained by 
concatenation? Several tricks are possible to overcome the problem, they are 
described in ijl.2.61 Using Proposition [51 we set ^ = (fii, fig, p) which has 
length 1^1 = \p\ + 0{\fii\ + ... + \fj,s\) — nH + O(logn) (Proposition [S] gives a 
much better bound but this is of no use here) . Then, u can be recovered from ^ 
which is a binary word of length nH + 0(log n). Thus, asymptotically, we get a 
better upper bound than n{H +1), the one given by Shannon for codings with 
prefix-free codes. 

Of course, ^ is no more obtained from u via a morphism (i.e. a map which 
preserves concatenation of words) between the monoid of words in alphabet A 
to that of binary words. 

1.2.6 Coding finitely many words with one word 

How can we code two words w, v by one word? The simplest way is to consider 
u%v where $ is a fresh symbol outside the alphabet of u and v. But what if 
we want to stick to binary words? As said above, the concatenation of u and 
V does not do the job: one cannot recover the right prefix u in uv. A simple 
trick is to also concatenate the length of |u| in unary and delimitate it by a zero: 
indeed, from the word one can recover u and v. In other words, the map 

(u,w) fl^lOuu is injective from {0, 1}* x {0, 1}* {0, 1}*. In this way, the 
code of the pair (u,w) has length 2\u\ + \v\ + 1. 
This can obviously be extended to more arguments. 

Proposition 5. Let s > 1. There exists a map { ) : ({0,1}*)^+^ {0,1}* 
which is injective and computable and such that, for all Ui, ...,Us,v G {0,1}*, 
\{ui,...,Us,v)\ = 2(|ui| + ... + |u,|) + +s. 

This can be improved, we shall need this technical improvement in SJ5T27TJ 

Proposition 6. There exists an injective and computable such that, for all 
ui,...,Us,v e {0,1}*, 

\{ui,...,Us,v)\ = (|ui| + ... + \us\ + \v\) + (log|ui| + ... + log|w^|) 

+ 0((loglog + ... + log log \Us\)) 

Proof. We consider the case s — 1, i.e. we want to code a pair {u,v). In- 
stead of putting the prefix l'"'o, let's put the binary representation /3(|ii|) 
of the number \u\ prefixed by its length. This gives the more complex 
code: ll''(l"l'l0/3(iui)itu with length 

\u\ + +2([log|u|J + 1) + 1 < l?tl + li;] +2 log lit] +3 

The first block of ones gives the length of /3(|m|). Using this length, we 
can get /3(|u|) as the factor following this first block of ones. Now, /3(|m|) 
is the binary representation of so we get |m| and can now separate u 
and V in the suffix uv. □ 
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1.3 Probabilistic approach 

The abstract probabilistic approach allows for considerable extensions of the 
results described in m.'2\ 

First, the restriction to fixed given frequencies can be relaxed. The probability 
of writing may depend on what has been already written. For instance, 
Shannon's source coding theorem has been extended to the so called "ergodic 
asymptotically mean stationary source models" . 

Second, one can consider a lossy coding: some length n words in alphabet A are 
ill-treated or ignored. Let i5 be the probability of this set of words. Shannon's 
theorem extends as follows: 

- whatever close to 1 is < 1, one can compress u only down to nH bits. 

- whatever close to is (5 > 0, one can achieve compression of u down to nH 
bits. 

1.4 Algorithmic approach 
1.4.1 Berry's paradox 

So far, we considered two kinds of binary codings for a word u in alphabet 
ai,...,as. The simplest one uses variable-length prefix- free codes ( §1.2.2p . The 
other one codes the rank of u as a member of some set ( §1.2.5p . 
Clearly, there are plenty of other ways to encode any mathematical object. Why 
not consider all of them? And define the information content of a mathematical 
object X as the shortest univoque description of x ( written as a binary word). 
Though quite appealing, this notion is ill defined as stressed by Berry's paradojo: 

Let P be the lexicographically least binary word which cannot be uni- 
voquely described by any binary word of length less than 1000. 

This description of (3 contains 106 symbols of written English (including spaces) 
and, using ASCII codes, can be written as a binary word of length 106 x 7 = 
742. Assuming such a description to be well defined would lead to a univoque 
description of (3 in 742 bits, hence less than 1000, a contradiction to the definition 
of p. 

The solution to this inconsistency is clear: the quite vague notion of univoque 
description entering Berry's paradox is used both inside the sentence describing 
p and inside the argument to get the contradiction. A collapse of two levels: 

- the would be formal level carrying the description of /3 

- and the meta level which carries the inconsistency argument. 

Any formalization of the notion of description should drastically reduce its scope 
and totally forbid the above collapse. 

^ Berry's paradox is mentioned by Bertrand Russell, 1908 ( I31| . p. 222 or 150) who credited 
G.G. Berry, an Oxford librarian, for the suggestion. 
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1.4.2 The turn to computability 

To get around the stumbling block of Berry's paradox and have a formal notion 
of description with wide scope, Andrei Nikolaievitch Kolmogorov (1903-1987) 
made an ingenious move: he turned to computability and replaced description 
by computation program. Exploiting the successful formalization of this a pri- 
ori vague notion which was achieved in the thirtie^. This approach was first 
announced by Kolmogorov in 1963, and then developped in [22, 1965. 

Similar approaches were also independently developped by Ray J. Solomonoff 
in [361 [32, 1964, and by Gregory Chaitin in f4','5], 1966-69. 

1.4.3 Digression on computability theory 

The formalized notion of computable function (also called recursive function) 
goes along with that of partial computable Junction which should rather be called 
partially computable partial Junction (also called partial recursive function), i.e. 
the partial qualifier has to be distributedEI 
So, there are two theories : 

- the theory oj computable Junctions^ 

- the theory oJ partial computable Junctions. 

The "right" theory, the one with a cornucopia of spectacular results, is that of 
partial computable functions. 

Let's pick up three fundamental results out of the cornucopia, which we 
state in terms of computers and programming languages. Let T and O be 
non empty finite products of simple countable families of mathematical 
objects such as N, A* (the family of words in alphabet A) where A is finite 
or countably infinite. 

Theorem 7. 1. [Enumeration theorem] The (program, input) ^ output 
function which executes programs on their inputs is itself partial com- 
putable. 

Formally, this means that there exists a partial computable function 

[/ : {0, 1}' X J ^ O 

such that the family of partial computable function I ^ O is exactly 
{;7e I e G {0, 1}*} where Ue{x) = U{e,x). 

Such a function U is called universal for partial computable functions 

2. [Parameter theorem (or s™ thm)]. One can exchange input and program 
(this is von Neumann's key idea for computers). 

Formally, this means that, letting T = Xi XX2, universal maps Ui-^^^x^ and 
o,re such that there exists a computable total map s : {0, 1}* x Ii —> 
{0, 1}* such that, for all e G {0, 1}*, xi G Ti and X2 G I2, 

t/iixi2(e> {xi,X2)) = Ui2{s{e,x\),X2) 

Through the works of Alonzo Church (via lambda calculus), Alan Mathison Turing (via 
Turing machines) and Kurt Godel and Jacques Herbrand (via Herbrand-Godel systems of 
equations) and Stephen Cole Kleene (via the recursion and minimization operators). 
■^In French, Daniel Lacombe used the expression semi-fonction semi-recursive 
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3. [Kleene fixed point tlieorem] For any transformation of programs, there 
is a program which does the same input — > output job as its transformed 
program. (Note: This is the seed of computer virology... cf. 2006) 
Formally, this means that, for every partial computable map f : {0, 1}* 
{0, 1}*, there exists e such that 

Ve e {0, 1}* Va; e J U{f{e),x) = U{e,x) 

1.5 Kolmogorov complexity and the invariance theorem 

Note. The denotations of (plain) Kolmogorov complexity and its prefix 
version may cause some confusion. They long used to be respectively 
denoted by K and H in the literature. But in their book [25], Li & 
Vitanyi respectively denoted them C and K. Due to the large success of 
this book, these last denotations are since used in many papers. So that 
two incompatible denotations now appear in the litterature. Since we 
mainly focus on plain Kolmogorov complexity, we stick to the traditional 
denotations K and H. 

1.5.1 Program size complexity or Kolmogorov complexity 

Turning to computability, the basic idea for Kohiiogorov complexity is 

description = program 

When we say "program", we mean a program taken from a family of programs, 
i.e. written in a programming language or describing a Turing machine or a 
system of Herbrand-Godel equations or a Post system,... 

Since we are soon going to consider length of programs, following what has been 
said in ill. 1.21 wc normalize programs: they will be binary words, i.e. elements 
of {0,1}*. 

So, we have to fix a function : {0, 1}* O and consider that the output of a 
program p is ^p{p). 

Which Lp are we to consider? Since we know that there are universal partial com- 
putable functions (i.e. functions able to emulate any other partial computable 
function modulo a computable transformation of programs, in other words, a 
compiler from one language to another), it is natural to consider universal par- 
tial computable functions. Which agrees with what has been said in i ll. 4. 31 
The general definition of the Kolmogorov complexity associated to any function 
{0,1}* -> O is as follows. 

Definition 8. //(/?: {0, 1}* O is a partial function, set : O ^ N 

K^{y) = min{|p| : ip{p) ^ y} 

Intuition: p is a program (with no input), Lp executes programs (i.e. Lp is all 
together a programming language plus a compiler plus a machinery to run pro- 
grams) and ip{p) is the output of the run of program p. Thus, for y Cz O, Kip{y) 
is the length of shortest programs p with which ip computes y (i.e. (p{p) = y) 
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As said above, we shall consider this definition for partial computable func- 
tions {0, 1}* — > O. Of course, this forces to consider a set O endowed with a 
computability structure. Hence the choice of sets that we shall call elementary 
which do not exhaust all possible ones but will suffice for the results mentioned 
in this paper. 

Definition 9. The family of elementary sets is obtained as follows: 

- it contains N and the A* 's where A is a finite or countable alphabet, 

- it is closed under finite (non empty) product, product with any non empty finite 
set and the finite sequence operator 



1.5.2 Tiie invariance theorem 

The problem with Definition [5] is that strongly depends on (p. Here comes a 
remarkable result, the invariance theorem, which insures that there is a smallest 
Kip up to a constant. It turns out that the proof of this theorem only needs 
the enumeration theorem and makes no use of the parameter theorem (usually 
omnipresent in computability theory). 

Tiieorem 10 (Invariance theorem, Kolmogorov, [52], 1965). Let O he an el- 
ementary set. Among the K^p's, where Lp : {0,1}* — > O varies in the family 
PC'-' of partial computable functions, there is a smallest one, up to an additive 
constant (= within some hounded interval). I.e. 

3V e PC'^ Vip e PC'^ BcVyeO Kv{y) < K,p{y) + c 

Such aV is called optimal. 

Proof. Let U : {0, 1}* x {0, 1}* ^ O be a partial computable universal 
function for partial computable functions {0, 1}* O (cf. Theorem [71 
Enumeration theorem). 

Let c : {0, 1}* x {0, 1}* {0, 1}* be a total computable injective map 
such that |c(e,a;)| = 2|e| + (cf. Proposition [5}. 
Define V : {0, 1}* ^ O as follows: 

Ve G {0, 1}* "ix G {0, 1}* V{c{e,x)) = U{e,x) 

where equality means that both sides are simultaneously defined or not. 
Then, for every partial computable function tp : {0, 1}* — > O, for every y £ 
O, if ip = Ue (i.e. p{x) = U(e,x) for all x, cf. Theorem [T] Enumeration 
theorem) then 

Kv{y) = least \p\ such that V{p) = y 

< least |c(e,a::)| such that V{c(e,x)) = y 
= least |c(e,a::)| such that U{e,x)) — y 
= least |x| + 2|e| + 1 such that p{x) — y 

since |c(e, a::)| = + 2|e| + 1 and p{x) = U (e, x) 
= (least |a;| such that p{x) = y) + 2|e| + 1 
= if^(y) + 2|e| + 1 

□ 
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Using the invariance theorem, the Kolmogorov complexity if : O ^ N is 
defined as Ky where V is any fixed optimal function. The arbitrariness of the 
choice of V does not modify drastically Ky, merely up to a constant. 

Definition 11. Kolmogorov complexity K'~' : O — > N is K^p where ip is some 
fixed optimal function T O. K'~^ will be denoted by K when O is clear from 
context. 

K*^ is therefore minimum among the K^p 's, up to an additive constant. 

K'^ is defined up to an additive constant: if V and V' are both optimal then 

3c Va; G O \Kv{x) - Kv'{x)\ < c 

1.5.3 About the constant 

So Kolmogorov complexity is an integer defined up to a constant. . . ! But the 

constant is uniformly bounded for x £ O. 

Let's quote what Kolmogorov said about the constant in [25]: 

Of course, one can avoid the indeterminacies associated with the 
[above] constants, by considering particular [. . .functions V], but it 
is doubtful that this can be done without explicit arbitrariness. 
One must, however, suppose that the different "reasonable" [above 
optimal functions] will lead to "complexity estimates" that will con- 
verge on hundreds of bits instead of tens of thousands. 
Hence, such quantities as the "complexity" of the text of "War and 
Peace" can be assumed to be defined with what amounts to unique- 
ness. 

In fact, this constant is in relation with the multitude of models of computa- 
tion: universal Turing machines, universal cellular automata, Her brand- Godel 
systems of equations. Post systems, KLeene definitions,... If we feel that one of 
them is canonical then we may consider the associated Kolmogorov complexity 
as the right one and forget about the constant. This has been developed for 
Schoenfinkel-Curry combinators S,K,I by Tromp [25] §3.2.2-3.2.6. 
However, this does absolutely not lessen the importance of the invariance the- 
orem since it tells us that K is less than any (up to a constant). A result 
which is applied again and again to develop the theory. 

1.5.4 Conditional Kolmogorov complexity 

In the enumeration theorem (cf. Theorem [7]), we considered (program, input) 
output functions. Then, in the definition of Kolmogorov complexity, we gave 
up the inputs, dealing with functions program output. 

Conditional Kolmogorov complexity deals with the inputs. Instead of measur- 
ing the information content of y G O, we measure it given as free some object 
z, which may help to compute y. A trivial case is when z = y, then the infor- 
mation content of y given y is null. In fact, there is an obvious program which 
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outputs exactly its input, whatever be the input. 

Let's mention that, in computer science, inputs are also considered as the envi- 
ronment. 

Let's give the formal definition and the adequate invariance theorem. 
Definition 12. // ip : {0, 1}* x I ^ O is a partial function, set \ ) : 

K^{y I z) = min{|p| | (p{p, z) = y} 

Intuition: p is a program (with no input), ip executes programs (i.e. is all 
together a programming language plus a compiler plus a machinery to run pro- 
grams) and ip{p, z) is the output of the run of program p on input z. Thus, for 
y G O, K^{y) is the length of shortest programs p with which if computes y on 
input z (i.e. <p{p,z) =y) 

Theorem 13 (Invariance theorem for conditional complexity). Among the 
I ) 's, where tp varies in the family PC^ of partial computable function 
{0, 1}* X X — > O, there is a smallest one, up to an additive constant (i.e. within 
some hounded interval) : 

3V G PCf yip G PCf 3c Vy G C G J Kv{y \ z) < K^{x \y) + c 

Such aV is called optimal. 

Proof. Simple application of the enumeration theorem for partial computable 
functions. □ 

Definition 14. : O X X — > N 15 Kv{ I ) where V is some fixed optimal 
function. 

Kj is defined up to an additive constant: if V et V' are both minimum then 

3c Vy G O G J \Kv{y \ z) - Kv'{y \ z)\ < c 

Again, an integer defined up to a constant. . . ! However, the constant is uniform 
in y G O and z €2. 

1.5.5 Simple upper bounds for Kolmogorov complexity 

Finally, let's mention rather trivial upper bounds: 

- the information content of a word is at most its length. 

- conditional complexity cannot be harder than the non conditional one. 

Proposition 15. 1. There exists c such that, 

Vx G {0, 1}* ii'^0'i>* (x) < |x| + c , Vn G N K^{n) < log(n) + c 

2. There exists c such that, 

\/x€D\/y€E K{x I y) < K{x) + c 

3. Letf:0^ O' be computable. Then, K^'{f{x)) < K^{x) + 0(1). 
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Proof. We only prove 1. Let Id : {0, 1}* ^ {0, 1}* be the identity 
function. The invariance theorem insures that there exists c such that 
j({o.ir < J<(o.i}* particular, for all x G {0,1}*, A'<°'^>'(a;) < 

\x\ +c. 

Let 9 : {0, 1}* ^ N be the function which associate to a word u = ttk-i-.-ao 
the integer 

e{u) = (2'= + afc_i2'=-^ + ... + 2ai + ao) - 1 

(i.e. the predecessor of the integer with binary representation 1m). Clearly, 
K^{n) — [log(n + 1)J . The invariance theorem insures that there exists c 
such that is:" < A'e + c. Hence K^^{n) < log(n) + c + 1 for all G N. □ 

The following property is a variation of an argument already used in ill. 2. 51 
the rank of an element in a set defines it, and if the set is computable, so is this 
process. 

Proposition 16. Let A C N x _D be computable such that An = Af) {{n} x D) 
is finite for all n. Then, letting tt(X) be the number of elements of X , 

3c Vx e An K{x I n) < log(tt(A„)) + c 

Intuition. An element in a set is determined by its rank. And this is a com- 
putable process. 

Proof. Observe that x is determined by its rank in An. This rank is an 
integer < tt^„ hence with binary representation of length < [log((jA„)J + 
1. □ 

1.6 Oracular Kolmogorov complexity 

As is always the case in computability theory, everything relativizes to any or- 
acle Z. This means that the equation given at the start of ^1.5l now becomes 

description — program of a partial Z-computable function 

and for each possible oracle Z there exists a Kolmogorov complexity relative to 
oracle Z. 

Oracles in computability theory can also be considered as second-order argu- 
ments of computable or partial computable functionals. The same holds with 
oracular Kolmogorov complexity: the oracle Z can be seen as a second-order 
condition for a second-order conditional Kolmogorov complexity 

K{y I Z) where if ( | ) : O x P{1) N 

Which has the advantage that the unavoidable constant in the "up to a con- 
stant" properties does not depend on the particular oracle. It depends solely 
on the considered functional. 

Finally, one can mix first-order and second-order conditions, leading to a condi- 
tional Kolmogorov complexity with both first-order and second-order conditions 

K{y I 2, Z) where K{ \ , ) : O x I x P{I) ^ N 

We shall see in i|4.2.3l an interesting property involving oracular Kolmogorov 
complexity. 
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2 Kolmogorov complexity and undecidability 



2.1 is unbounded 

Let K ^ Kv ■■ O ~> N where V : {0, 1}* ^ O is optimal (cf. Theorem gTO]). 
Since there are finitely many programs of size < n (namely 2""*"^ — 1 words), 
there are finitely many elements of O with Kolmogorov complexity less than n. 
This shows that K is unbounded. 

2.2 is not computable 

Berry' paradox (cf. i|1.4.1[) has a counterpart in terms of Kolmogorov complex- 
ity, namely it gives a proof that K, which is a total function O ^ N, is not 
computable. 

Proof. For simplicity of notations, we consider the case O = N. Define 
L : N ^ O as follows; 

Z/(n) — least k such that K{k) > 2n 

So that K{L{n)) > n for all n. If K were computable so would be L. Let 
V : O ^ N be optimal, i.e. K — Ky- The invariance theorem insures 
that there exists c such that K < Kl + c. Observe that Kl^LIji) < n by 
definition of Kl- Then 

2n < K{L{n)) < KL{L{n) + c < n + c 

A contradiction for n > c. □ 

The undecidability of K can be sen as a version of the undecidability of the 
halting problem. In fact, there is a simple way to compute K when the halting 
problem is used as an oracle. To get the value of K{x), proceed as follows: 

- enumerate the programs in {0, 1}* in lexicographic order, 

- for each program p check if V{p) halts (using the oracle), 

- in case V{p) halts then compute its value, 

- halt and output \p\ when some p is obtained such that V{p) — x. 

The argument for the undecidability of K can be used to prove a much stronger 
statement: K can not be bounded from below by an unbounded partial com- 
putable function. 

Theorem 17 (Kolmogorov). There is no unbounded partial recursive function 
: O ^ N such that ip{x) < K(x) for all x in the domain of Lp. 

Of course, K is bounded from above by a total computable function, cf. 
Proposition [T^ 

2.3 K is computable from above 

Though K is not computable, it can be approximated from above. The idea 
is simple. Suppose O = {0, 1}*. consider all programs of length less than |a;| 
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and let them be executed during t steps. If none of them converges and outputs 
X then the t-bound is If some of them converges and outputs x then the 
bound is the length of the shortest such program. 

The limit of this process is K{x), it is obtained at some finite step which we are 
not able to bound. 

Formally, this means that there is some F : O x N ^ N which is computable 
and decreasing in its second argument such that 

K{x) = lim F{x,n) 

n — ^+00 

2.4 Kolmogorov complexity and Godel's incompleteness 
theorem 

Godel's incompleteness' theorem has a striking version, due to Chaitin, 1971-74 
[51 [7] , in terms of Kolmogorov complexity. In the language of arithmetic one 
can formalize partial computability (this is Godel main technical ingredient for 
the proof of the incompleteness theorem) hence also Kolmogorov complexity. 
Chaitin proved an n lower bound to the information content of finite families 
of statements about finite restrictions associated to an integer n of the halting 
problem or the values of K. 

In particular, for any formal system T, if n is bigger than the Kolmogorov 
complexity of T (plus some constant, independent of T) such statements cannot 
all be provable in T 

Theorem 18 (Chaitin, 1974 [7]). Suppose O = {0, 1}*. 

1. Let V : {0, 1}* O be optimal (i.e. K = Ky). Let Tn be the family of true 
statements 3p {V{p) — x) for \x\ < n (i.e. the halting problem for V limited to 
the finitely many words of length < n). Then there exists a constant c such that 
K{Tn) > n ~ c for all n. 

2. Let Tn be the family of true statements K{x) > \x\) for \x\ < n. Then there 
exists a constant c such that K{Tn) > n ^ c for all n. 

Note. In the statement of the theorem, K{x) refers to the Kolmogorov com- 
plexity on O whereas K(Tn) refers to that on an adequate elementary family 
(cf. Definitional). 

3 Formalization of randomness for finite objects 

3.1 Probabilities: laws about a non formalized intuition 

Random objects (words, integers, reals,...) constitute the basic intuition for 
probabilities . . . but they are not considered per se. No formal definition of ran- 
dom object is given: there seems there is no need for such a formal concept. The 
existing formal notion of random variable has nothing to do with randomness: 
a random variable is merely a measurable function which can be as non random 
as one likes. 
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It sounds strange that the mathematical theory which deals with randomness 
removes the natural basic questions: 

- what is a random string? 

- what is a random infinite sequence? 

When questioned, people in probability theory agree that they skip these ques- 
tions but do not feel sorry about it. As it is, the theory deals with laws of 
randomness and is so successful that it can do without entering this problem. 

This may seem to be analogous to what is the case in geometry. What are 
points, lines, planes? No definition is given, only relations between them. Giv- 
ing up the quest for an analysis of the nature of geometrical objects in profit of 
the axiomatic method has been a considerable scientific step. 
However, we contest such an analogy. Random objects are heavily used in many 
areas of science and technology: sampling, cryptology,... Of course, such objects 
are are in fact "as much as we can random". Which means fake randomness. 

Anyone who considers arithmetical methods of producing random 
reals is, of course, in a state of sin. For, as has been pointed out 
several times, there is no such thing as a random number — there are 
only methods to produce random numbers, and a strict arithmetical 
procedure is of course not such a method. 

John von Neumann, 1951 ^\ 

So, what is "true" randomness? Is there something like a degree of randomness? 
Presently, (fake) randomness only means to pass some statistical tests. One can 
ask for more. 

In fact, since Pierre Simon de Laplace (1749-1827), some probabilists never gave 
up the idea of formalizing the notion of random object. Let's cite particularly 
Richard von Mises (1883-1953) and Kolmogorov. In fact, it is quite impressive 
that, having so brilliantly and efficiently axiomatized probability theory via 
measure theory in 1933 [20], Kolmogorov was not fully satisfied of such founda- 
tions!^ And kept a keen interest to the quest for a formal notion of randomness 
initiated by von Mises in the 20 's. 

3.2 The 100 heads paradoxical result in probability theory 

That probability theory fails to completely account for randomness is strongly 
witnessed by the following paradoxical fact. In probability theory, if we toss 
an unbiaised coin 100 times then 100 heads are just as probable as any other 
outcome! Who really believes that ? 

The axioms of probability theory, as developped by Kolmogorov, do 
not solve all mysteries that they are sometimes supposed to. 

Peter Gdcs [I^/ 

^ Kolmogorov is one of the rare probabilists - up to now — not to believe that Kolmogorov's 
axioms for probability theory do not constitute the last word about formalizing randomness... 
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3.3 Kolmogorov's proposal: incompressible strings 

We now assume that O = {0,l}*,i.e. we restrict to words. 



3.3.1 incompressibility with Kolmogorov complexity 

Though much work has been devoted to get a mathematical theory of random 
objects, notably by von Mises [351 [3S]i none was satisfactory up to the 60's 
when Kolmogorov based such a theory on Kolmogorov complexity, hence on 
computability theory. 

The theory was, in fact, independently developed by Gregory J. Chaitin (b. 
1947), 1966 H, 1969 [5] (both papers submitted in 1965)0 

The basic idea is as follows: larger is the Kolmogorov complexity of a text, more 
random is this text, larger is its information content, and more compressed is 
this text. 

Thus, a theory for measuring the information content is also a theory of ran- 
domness. 

Recall that there exists c such that for all x G {0, 1}*, K{x) < |x| + c (Proposi- 
tionlini). Also, there is a "stupid" program of length about |a;| which computes 
the word x : tell the successive letters of x. The intuition of incompressibility 
is as follows: x is incompressible if ther no shorter way to get x. 
Of course, we are not going to define absolute randomness for words. But a 
measure of randomness based on how far from |a::| is K{x). 

Definition 19 (Measure of incompressibility). 

A word X is c-incompressible if K{x) > \x\ — c. 

As is rather intuitive, most things are random. The next Proposition for- 
malizes this idea. 

Proposition 20. The proportion of c-incompressible strings of length n is > 
1 - 2"^ 

Proof. At most 2""^^ — 1 programs of length < n — c and 2" strings of 
length n. □ 

3.3.2 incompressibility with length conditional Kolmogorov com- 
plexity 

We observed in iil.2.3l that the entropy of a word of the form 000. ..0 is null. I.e. 
entropy did not considered the information conveyed by the length. 
Here, with incompressibility based on Kolmogorov complexity, we can also ig- 
nore the information content conveyed by the length by considering incompress- 
ibility based on length conditional Kolmogorov complexity. 

Definition 21 (Measure of length conditional incompressibility). A word x is 
length conditional c-incompressible if K{x \ \x\) > \x\ — c. 

^ For a detailed analysis of who did what, and when, see |25) p. 89— 92. 
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The same simple counting argument yields the following Proposition. 

Proposition 22. The proportion of length conditional c-incompressible strings 
of length n is > 1 — 2""^. 

A priori length conditional incompressibility is stronger than mere incom- 
pressibility. However, the two notions of incompressibility are about the same 
. . . up to a constant. 

Proposition 23. There exists d such that, for all c E N and x E {0, 1}* 

1. X is length conditional c-incompressible ^ x is {c + d) -incompressible 

2. X is c-incompressible ^ x is length conditional {2c + d) -incompressible. 

Proof. 1 is trivial. For 2, observe that there exists e such that, for all x, 

(*) K{x) < K{x I |x|) + 27^(1x1 - K{x \ \x\)) + d 

In fact, ii K = and K{ \ ) — K^(^ | ) and 

1x1 — K{x I \x\) — ifiip) i'iq I l^l) — x 
K{\x\ — K{x I |a;|)) = \p\ K{x \ \x\) = |q| 

With p and q, hence with {p,q) (cf. Proposition [5]) , one can successively 

\x\ — K{x I |a;|) this is <^(p) 
K{x I \x\) this is q 

\x\ just sum 

X this is ^(q \ \x\) 

Using K < log+ci and K{x) > \x\ — c, (*) yields 



get < 



\x\ - K(x I < 21og(|x| - K{x \ |x|)) + 2ci +c + d 
Finally, observe that z <2 log 2 + insures z < max(8, 2k). □ 

3.4 Incompressibility is randomness: Martin-L6f 's argu- 
ment 

Now, if incompressibility is clearly a necessary condition for randomness, how do 
we argue that it is a sufficient condition? Contraposing the wanted implication, 
let's see that if a word fails some statistical test then it is not incompressible. 
We consider some spectacular failures of statistical tests. 

Example 24. 1. [Constant left half length prefix] For all n large enough, a 
string 0"m with \u\ — n cannot be c-incompressible. 

2. [Palindromes] Large enough palindromes cannot be c-incompressible. 

3. [0 and 1 not equidistributed] For all < a < 1, for all n large enough, a 
string of length n which has < aj zeros cannot be c-incompressible. 
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Proof. 1. Let c' be such that K{x) < \x\ + c'. Observe that there exists 
c" such that K{0"u) < K(u) + c" hence 

Ki^Tu) < n + c' + c" < \\Q"u\ + c' + c" 

So that K(Q^u) > |0"ii| — c is impossible for n large enough. 

2. Same argument; There exists c" such that, for all palindrome x, 

K{x)<^\x\+c" 



3. The proof follows the classical argument to get the law of large numbers 
(cf. Feller's book [IB]). Let's do it for a = |, so that f = |. 

Let An be the set of strings of length n with < ^ zeros. We estimate the 
number of elements of An. 



, n 



i — 6 6 



Use Stirling's formula (1730) 



'271 TT 



/n\" 1 , /- — j_ 

ei2"+i < n! < V2n7r (^-j ei2" 



Using Proposition 1161 for any element of An, we have 

i^(a;|n)<log(iV)+d<nlog(^-|=^ + + d 

Since ^ < 8, we have -|= < 2 and log {j-^i^ < 1- Hence, n — c < 

nlog (^"^) + ^^2^ + d is impossible for n large enough. 

So that X cannot be c-incompressible. □ 

Let's give a common framework to the three above examples so as to get 
some flavor of what can be a statistical test. To do this, we follow the above 
proofs of compressibility. 

Example 25. 1. [Constant left half length prefix] 

Set Vm — 0,11 strings with m zeros ahead. The sequence Vq, Vi, ... is decreasing. 
The number of strings of length n in Vm is if m > n and 2"^™ if m < n. 
Thus, the proportion Hj^lkl-'^^^'^ xev^} i^j^gf]^ ^ words which are in Vm is 
2~m 

2. [Palindromes] Put in Vm all strings which have equal length m prefix and 
suffix. The sequence Vo,Vi,... is decreasing. The number of strings of length n 
in Vm is if m > 2. and 2"~^™ if m < ^. Thus, the proportion of length n 
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words which are in Vm is 2 

3. [0 and 1 not equidistributed] Put in V"^ = all strings x such that the number 
of zeros is < (a + (1 - 0)2-") The sequence Vo,Vi,... is decreasing. A 
computation analogous to that done in the proof of the law of large numbers 
shows that the proportion of length n words which are in Vm is < 2^'^™' for 
some 7 > (independent of m). 

Now, what about other statistical tests? But what is a statistical test? A 
convincing formalization has been developed by Martin-L6f. The intuition is 
that illustrated in Example [25l augmented of the following feature: each Vm is 
computably enumerable and so is the relation {{m,x) \ x G Vm}- A feature 
which is analogous to the partial computability assumption in the definition of 
Kolmogorov complexity. 

Definition 26. [Abstract notion of statistical test, Martin-Lof 1964] ^ statis- 
tical test is a family of nested critical regions 

{0,1}* 3 ^ ■^i ...2Vm2 - 

such that {{m,x) \ x G Vm} is computably enumerable and the proportion 
•i{x\\x\—n^A xeVm} i^^gf^f^ words which are in Vm is 2~™. 

Intuition. The bound 2^™ is just a normalization. Any bound h{n) such that 
6 : N — > Q which is computable, decreasing and with limit could replace 2~™. 
The significance of x & Vm is that the hypothesis x is random is rejected with 
significance level 2~™. 

Remark 27. Instead of sets Vm one can consider a function 5 : {0, 1}* N such 
that ti^lkl-'^^A x£V^} ^ a,nd 5 is computable from below, i.e. {(m, a;) 
5{x) > to} is recursively enumerable. 

We have just argued on some examples that all statistical tests from practice 
are of the form stated by Definition [26l Now comes Martin-Lof fundamental 
result about statistical tests which is in the vein of the invariance theorem. 

Theorem 28 (Martin-Lof, 1965). Up to a constant shift, there exists a largest 
statistical test {Um)meN 

V(Ki)meN 3c Vm Vn+c ^ Um 

In terms of functions, up to an additive constant, there exists a largest statistical 
test A 

V(5 3c Vx 5{x) < A{x) + c 

Proof. Consider A(a;) — \x\ — K(x \ \x\) — 1. 

Clearly, {{rn,x) \ A(a;) > m} is computably enumerable. 



A is a test. 



A{x) > m means K{x \ \x\) < — m — 1. So no more elements in 
{a:: | A(3;) > m A Ix] = n} than programs of length < n — m — 1, which 
is 2"-™ - 1. 
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A is largest, x is determined by its rank in the set Vs(x) ~ {z \ S{z) > 

5{x) A \z\ = \x\}. Since this set has < 2"~'''^' elements, the rank of x has 
a binary representation of length < |a;j — S{x). Add useless zeros ahead to 
get a word p with length — 5{x). 

With p we get \x\ — S{x). With ja;| ^S{x) and \x\ we get S{x) and construct 
Vs(x)- With p we get the rank of x in this set, hence we get x. Thus, 
K{x 1 1^1) < \x\ - S{x) + c, i.e. S{x) < A{x) + c. □ 

The importance of the previous result is the following corollary which insures 
that, for words, incompressibility implies (hence is equivalent to) randomness. 

Corollary 29 (Martin-Lof, 1965). Incompressibility passes all statistical tests. 
I.e. for all c, for all statistical test {Vrn,)m, there exists d such that 

Va; (x is c-incompressible => a; ^ K:+(j) 

Proof. Let x be length conditional c-incompressible. This means that 
K{x I \x\) > \x\ — c. Hence A(a;) = \x\ — K{x \ \x\) — 1 < c — 1, which 
means that x ^Uc- 

Let now (Vin)m be a statistical test. Then there is some d such that 
Vm+d C Um Therefore x ^ Vc+d- □ 

Remark 30. Observe that incompressibility is a bottom-up notion: we look at 
the value of K{x) (or that of K{x \ \x\)). 

On the opposite, passing statistical tests is a top-down notion. To pass all 
statistical tests amounts to an inclusion in an intersection: namely, an inclusion 
in 

n u ^'"+^ 



3.5 Randomness: a new foundation for probability the- 
ory? 

Now that there is a sound mathematical notion of randomness (for finite ob- 
jects), or more exactly a measure of randomness, is it possible/reasonable to 
use it as a new foundation for probability theory? 

Kolmogorov has been ambiguous on this question. In his first paper on the 
subject (1965, [22], p. 7), Kolmogorov briefly evoked that possibility : 

... to consider the use of the [Algorithmic Information Theory] con- 
structions in providing a new basis for Probability Theory. 

However, later (1983, [23J, p. 35-36), he separated both topics 

"there is no need whatsoever to change the established construction 
of the mathematical probability theory on the basis of the general 
theory of measure. I am not enclined to attribute the significance 
of necessary foundations of probability theory to the investigations 
[about Kolmogorov complexity] that I am now going to survey. But 
they are most interesting in themselves. 
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though stressing the role of his new theory of random objects for mathematics 
as a whole ([23], p. 39): 

The concepts of information theory as apphed to infinite sequences 
give rise to very interesting investigations, which, without being in- 
dispensable as a basis of probability theory, can acquire a certain 
value in the investigation of the algorithmic side of mathematics as 
a whole. 

4 Formalization of randomness for infinite ob- 
jects 

We shall stick to infinite sequences of zeros and ones: {0, 1}^. 

4.1 Martin-Lof approach with topology and comput ability 

This approach is an extension to infinite sequences of the one he developed for 
finite objects, cf. §3.41 

To prove a probability law amounts to prove that a certain set X of sequences 
has probability one. To do this, one has to prove that the complement set 
Y = {0, 1}^ \ X has probability zero. Now, in order to prove that Y C {0, 1}^ 
has probability zero, basic measure theory tells us that one has to include Y in 
open sets with arbitrarily small probability. I.e. for each n S N one must find 
an open set Un ^ Y which has probability < 

If things were on the real line R we would say that ?7„ is a countable union of 
intervals with rational endpoints. 

Here, in {0, l}'^, [/„ is a countable union of sets of the form u{0, 1}^ where u is 
a finite binary string and u{0, 1}^ is the set of infinite sequences which extend 
u. 

In order to prove that Y has probability zero, for each n S N one must find a fam- 
ily {Un,7n)nieN SUCh that Y C\J^ Un,m{0, 1}" and Proba{\J^ Mn,m{0, f }^) < ^ 

for each n G N. 

Now, Martin-Lof makes a crucial observation: mathematical probability laws 
which we can consider necessarily have some effective character. And this ef- 
fectiveness should reflect in the proof as follows: the doubly indexed sequence 
{un,m)n,meN *■« Computable. 

Thus, the set [J^Un,m{0,iy^ is a computably enumerable open set and 
Hn Um ^n,m{0, 1}^ Is & countable intersection of a computably enumerable fam- 
ily of open sets. 

Now comes the essential theorem, which is completely analog to Theorem [28l 

Theorem 31 (Martin-Lof [26]). Let's call constructively null Gs set any set of 
the form f^^ [J^ Mn,m{0, f }^ where the sequence Un^m is computably enumerable 
and Pro5a(lJ^ u„_m{0, 1}^) < ^ (which implies that the intersection set has 
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probability zero). 

There exist a largest constructively null Gg set 

Let's insist that the theorem says largest, up to nothing, reaUy largest. 

Definition 32 (Martin-L6f [26]). A sequence a G {0, 1}^ is random if it belongs 
to no constructively null Gs set (i.e. if it does not belongs to the largest one). 

In particular, the family of random sequence, being the complement of a 
constructively null Gs set, has probability 1. 

4.2 The bottom-up approach 
4.2.1 The naive idea badly fails 

The natural naive idea is to extend randomness from finite objects to infinite 
ones. The obvious first approach is to consider sequences a £ {0, 1}^ such that, 
for some c, 

Vn K{a \n) > n — c (1) 
However, Martin-L6f proved that there is no such sequence. 

Theorem 33 (Martin-L6f [27]). For every a G {0, 1}^ there are infinitely many 
k such that K{a \ k) <k — logfc — 0(1). 

Proof, Let f{z) — k — ([logzj + 1) First, observe that 

f{z+2yf{z) = 2- Llog(z+2)J + [log 2j = 2- ( [log 2+log(l+^J - [log z]) > 
since log(l + | < 1 for kz > 1. 

Fix any m and consider a \ m. This word is the binary representation 
of some integer k such that m — [logfcj + 1. Now, consider x — a \ k 
and let y be the suffix of x of length k — m = f{k). From y we get 
\y\ = k — m = f{k). Since f{z + 2) — f{z) > 0, there are at most two 
(consecutive) integers k such that f{k) = \y\. One bit of information tells 
which one in case there are two of them. So, from y (plus one bit of 
information) one gets m. Hence the binary representation of m, which is 
a \m. By concatenation with y, we recover x = a\k. 
This process being effective. Proposition 1151 (point 3) insures that 

K{a\k) < K{y) + 0{1) < \y\ + 0{1) = fc-m + 0(l) = fc - fog fc + 0(1) 

□ 

The above argument can be extended to prove a much more general result. 

Theorem 34 (Large oscillations, Martin-L6f, 1971 i27J). Let f : N ^ N he 

a total computable function satisfying X^neN^"^'"' ~ +oo- Then, for every 
a € {0, 1}^, there are infinitely many k such that K{a f fc) < fc — /(fc). 
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4.2.2 Miller & Yu's theorem 



It took about forty years to get a characterization of randomness via plain 
Kolmogorov complexity which completes very simply Theorem 1341 

Theorem 35 (Miller & Yu, 2004 [15]). 1. Let f -.N^N be a total computable 
function satisfying X^nGN 2"^^"'' < +oo. Then, for every random a G {0,1}^, 
there exists c such that K{a \ k \ k) > k — f{k) — c for all k. 2. There exists 

a total computable function / : N ^ N satisfying X^neN 2^^*-"-' < +oo such 
that for every non random a G {0, 1}^ there are infinitely many k such that 
K{a\k) <k- f{k). 

Recently, an elementary proof of this theorem was given by Bienvenu & 
Merkle & Shen, [5]. 

4.2.3 Kolmogorov randomness and 0' 

A natural question following Theorem [33] is to look at the so-called Kolmogorov 
random sequences which satisfy K[a \ k) > k — 0{\) for infinitely many fc's. 
This question got a very surprising answer involving randomness with oracle 
the halting problem 0'. 

Theorem 36 (Nies, Stephan & Terwijn [3D]). Let a e {0,1}'^. There are 
infinitely many k such that K{a \ k) < k — f{k) (i.e. a is Kolmogorov random) 
if and only if a is ^' -random. 

4.2.4 Variants of Kolmogorov complexity and randomness 

Bottom-up characterization of random sequences were obtained by Chaitin, 
Levin and Schnorr using diverse variants of Kolmogorov complexity. . 

Definition 37. 1. [Schnorr, fS^ 1971] For O = {0, 1}*, the process complexity 
S is the variant of Kolmogorov complexity obtained by restricting to partial 
computable functions {0,1}* {0,1}* which are monotonous, i.e. if p is a 
prefix of q and V{p),V{q) are both defined then V{p) is a prefix ofV{q). 

2. [Chaitin, 1975] The prefix- free variant H of Kolmogorov complexity is 
obtained by restricting to partial computable functions {0, 1}* {0, 1}* which 
have prefix-free domains. 

3. [Levin, \40j] 1970] For O = {0, 1}*, the monotone variant Km of Kolmogorov 
complexity is obtained as follows: Kmix) is the least \p\ such that x is a prefix 
of U{p) where U is universal among monotone partial computable functions. 

Theorem 38. Let a £ {0,1}^. The following conditions Then a is random if 
and only if S{a \k) >k — 0(1) if and only if H{a \ k) > k — 0(1) if and only if 
Km{a\k) >k-0{l). 
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The main problem with these variants of Kolmogorov complexity is that 
there is no solid understanding of what the restrictions they involve really mean. 
Chaitin has introduced the idea of self-delimitation for prefix-free functions: 
since a program in the domain of U has no extension in the domain of U, it 
somehow know where it ends. Though interesting, this interpretation is not a 
definitive explanation as Chaitin himself admits (personal communication). 
Nevertheless, these variants have wonderful properties. Let's cite one of the 
most striking one: taking = N, the series 2~^'^"^ converges and is the biggest 
absolutely convergent series up to a multiplicative factor. 

5 Application of Kolmogorov complexity to clas- 
sification 

5.1 What is the problem? 

Striking results have been obtained, using Kolmogorov complexity, with the 
problem of classifying quite diverse families of objects: let them be literary 
texts, music pieces, examination scripts (lax supervised) or, at a different level, 
natural languages, natural species (philogeny). 

The authors, mainly Bennett, Vitanyi, Cilibrasi,.. have worked out refined 
methods which are along the following lines. 

(1) Define a specific family of objects which we want to classify. 

For example a set of Russian literary texts that we want to group by 
authors. In this simple case all texts are written in their original Russian 
language. Another instance, music. In that case a common translation is 
necessary, i.e. a normalization of the texts of these music pieces that we 
want to group by composer. This is required in order to be able to compare 
them. An instance at a different level: the 52 main european languages. In 
that case one has to choose a canonical text and its representations in each 
one of the different languages (i.e. corpus) that we consider. For instance, 
the Universal Declaration of Human Rights and its translations in these 
languages, an example which was a basic test for Vitanyi's method. As 
concerns natural species, the canonical object will be a DNA sequence. 
What has to be done is to select, define and normalize a family of objects 
or corpus that we want to classify. 
Observe that this is not always an obvious step: 

• There may be no possible normalization. For instance with artists 
paintings,. 

• The family to be classified may be finite though ill defined or even of 
unknown size, cf. 15.3.11 

(2) In fine we are with a family of words on some fixed alphabet representing 
objects for which we want to compare and measure pairwise the common 
information content. 
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This is done by defining a distance for these pairs of (binary) words with 
the following intuition: 

the more common information there is between two words, the 
closer they are and the shorter is their distance. Conversely, the 
less common information there is between two words, the more 
they are independent and non correlated, and bigger is their 
distance. 

Two identical words have a null distance. Two totally indepen- 
dent words (for example, words representing 100 coin tossing) 
have distance about 1 (for a normalized distance bounded by 
1)- 

Observe that the authors follow Kolmogorov's approach which was to 
define a numerical measure of information content of words, i.e. a measure 
of their randomness. In exactly the same way, a volume or a surface gets 
a numerical measure. 

(3) Associate a classification to the objects or corpus defined in (1) using the 
numerical measures of the distances introduced in (2). 

This step is presently the least formally defined. The authors give rep- 
resentations of the obtained classifications using tables, trees, graphs,... 
This is indeed more a visualization of the obtained classification than a 
formal classification. Here the authors have no powerful formal framework 
such as, for example, Codd's relational model of data bases and its exten- 
sion to object data bases with trees. How are we to interpret their tables 
or trees? We face a problem, a classical one. for instance with distances 
between DNA sequences. Or with the acyclic graph structure of Unix files 
in a computer. 

This is much as with the rudimentary, not too formal, classification of 
words in a dictionary of synonyms. 

Nevertheless, Vitanyi & al. obtained by his methods a classification tree 
for the 52 European languages which is that obtained by linguists, a re- 
markable success. And the phylogenetic trees relative to parenthood which 
are precisely those obtained via DNA sequence comparisons by biologists. 

(4) An important problem remains to use a distance to obtain a classification 
as in (3). Let's cite Cilibrasi [3]: 

Large objects (in the sense of long strings) that differ by a tiny 
part are intuitively closer than tiny objects that differ by the 
same amount. For example, two whole mitochondrial genomes 
of 18, 000 bases that differ by 9, 000 are very different, while two 
whole nuclear genomes of 3 x 10^ bases that differ by only 9, 000 
bases are very similar. Thus, absolute difference between two 
objects does not govern similarity, but relative difference seems 
to. 
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As we shall see, this problem is easy to fix by some normalization of 
distances. 

(5) Finally, all these methods rely on Kolmogorov complexity which is a non 
computable function (cf. H2.'2\i . The remarkable idea introduced by Vi- 
tanyi is as follows: 

• consider the Kolmogorov complexity of an object as the ultimate and 
ideal value of the compression of that object, 

• and compute approximations of this ideal compression using usual 
efficient compressors such as gzip, bzip2, PPM,... 

Observe that the quality and fastness of such compressors is largely due 
to heavy use of statistical tools. For example, PPM (Prediction by Par- 
tial Matching) uses a pleasing mix of statistical models arranged by trees, 
sufhx trees or suffix arrays. The remarkable efficiency of these tools is 
of course due to several dozens of years of research in data compression. 
And as time goes on, they improve and better approximate Kolmogorov 
complexity. 

Replacing the "pure' but non computable Kolmogorov complexity by a 
banal compression algorithm such as gzip is quite a daring step took by 
Vitanyi! 



5.2 Classification via compression 

5.2.1 The normalized information distance NID 

We now formalize the notions described above. The idea is to measure the 
information content shared by two binary words representing some objects in a 
family we want to classify. 

The first such tentative goes back to the 90's [T]: Bennett and al. define a 
notion of information distance between two words x,y as the size of the shortest 
program which maps x to y and y to x. These considerations rely on the notion 
of reversible computation. A possible formal definition for such a distance is 

ID{x,y) — least \p\ such that U{p,x) ~ y and U{p,y) — x 

where U : {0, 1}* x {0, 1}* {0, 1}* is optimal for K{ \ ). 
An alternative definition is as follows: s 

ID'{x,y) = ma,yi{K{x\y),K{y\x)] 

The intuition for these definitions is that the shortest program which computes 
X from y takes into accoulnt all similarities between x and y. 
Observe that the two definitions do not coincide (even up to logarithmic terms) 
but lead to similar developments and efficient applications. 
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Note. In the above formula, K can be plain Kolmogorov complexity or its prefix 
version. In fact, this does not matter for a simple reason: all properties involving 
this distance will be true up to a 0(log(|x|), log(|y|)) term and the difference 
between K{z\t) and H{z\t) is bounded by 21og(|z|). For conceptual simphcity, 
we stick to plain Kolmogorov complexity. 

ID and ID' satisfy the axioms of a distance up to a logarithmic term. The 
strict axioms for a distance d are 

d(x,x) = (identity) 
d{x,y) = d{y,x) (symmetry) 
d{x, z) < d{x, y) + d(y, z) (triangle inequality) 

The up to a log term axioms which are satisfied by ID and ID' are as follows: 

ID{x,x) = 0(1) 
ID{x,y) — ID{y,x) 

ID{x,z) < ID{x,y) + ID{y,z) + 0{log{ID{x,y) + ID{y,z))) 

Proof. Let e be such that U{e,x) = x for all x. Then ID(x,x) < jej — 
0(1). No better upper bound is possible (except if we assume that the 
empty word is such an e). 

Let now p,p' ,q,q' be shortest programs such that U(p,y) — x, U{p',x) = 
y, U{q,z) = y, U{q',y) = z. Thus, K{x\y) = \p\, K{y\x) = \p'\, K(y\z) = 

111 K{^\y) = 

Consider the injective computable function ( ) of Proposition [6] which is 
such that \{r,s)\ = \r\ + \s\ +0(log|r|). 

Set ip : {0, 1}* X {0, 1}* ^ {0, 1}* be such that ip{{r,s),x) = U{.s,U{r,x)). 
Then 

^{{q, p),z) = U {p, U{q,z)) = U (p, y) = x 
so that, by the invariance theorem, 

K{x\z) < K^{x\z) + 0(1) <\{q,p)\+ 0(1) 

= |g| + IpI + 0(log(|g|)) = K{y\z) + K{x\y) + 0{\og{K(y\z))) 

And similarly for the other terms. Which proves the stated approxima- 
tions of the axioms. □ 

It turns out that such approximations of the axioms are enough for the 
development of the theory. 

As said in H5.ll to avoid scale distortion, this distance ID is normalized to NID 
(normalized information distance) as follows: 

AT/nf^ ^ _ T^&AK{x\y),K{y\x)) 
iMU[x,y)- ^^^(^(^)^^(y)) 

The remaining problem is that this distance is not computable since K is not. 
Here comes Vitanyi's daring idea: consider this NID as an ideal distance which 
is to be approximated by replacing the Kolmogorov function K by computable 
compression algorithms which go on improving. 
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5.2.2 The normalized compression distance NCD 

The approximation of K{x) by C{x) where C is a compressor, does not sufhee. 
We also have to approximate the conditional Kolmogorov complexity K{x\y). 
Vitanyi chooses the following approximation: 

C{y\x) = C{xy) - C\x) 

The authors explain as follows their intuition. 
To compress the word xy {x concatenated to y), 

- the compressor first compresses x, 

- then it compresses y but skip all information from y which was already in x. 
Thus, the output is not a compression of y but a compression of y with all x 
information removed. I.e. this output is a conditional compression of y knowing 

X. 

Now, the assumption that the compressor first compresses x is questionable: 
how does the compressor recovers x in xy 1. One can argue positively in case 
x, y are random (= incompressible) and in case x = y. And between these two 
extreme cases? But it works... The miracle of modelization? Or something not 
completely understood? 

With this approximation, plus the assumption that C{xy) = C{yx) (also ques- 
tionable) we get the following approximation of NID, called the normalized 
compression distance NCD : 

msyi{C{x\y),C{y\x)) 

max(C(a;),C(y)) 
ma.^{C{yx) - C{y),C{xy) - C{x)) 

max(C(x), C{y)) 
C{xy)-imn{C{x),C{y)) 
maj,{C{x),C{y)) 

Clustering according to NCD and, more generally, classification via compres- 
sion, is a kind of black box: words are grouped together according to features 
that are not explicitly known to us. Moreover, there is no reasonable hope that 
the analysis of the computation done by the compressor gives some light on the 
obtained clusters. For example, what makes a text by Tolstoi so characteristic? 
What differentiates the styles of Tolstoi' and Dostoievski? But it works, Russian 
texts are grouped by authors by a compressor which ignores everything about 
Russian literature. 

When dealing with some classification obtained by compression, one should 
have some idea of this classification: this is semantics whereas the compressor 
is purely syntactic and docs not understand an}d;hing. 

This is very much like with machines which, given some formal deduction sys- 
tem, are able to prove quite complex statements. But these theorems are proved 
with no explicit semantical idea, how are we to interpret them? No hope that 
the machine gives any hint. 



NCD{x,y) = 
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5.3 The Google classification 



Though it does not use Kolmogorov complexity, we now present another recent 
approach by Vitanyi and Cilibrasi fTD to classification which leads to a very 
performing tool. 

5.3.1 The normalized Google distance NGD 

This quite original method is based on the huge data bank constituted by the 
world wide web and the Google search engine which allows for basic queries 
using conjunction of keywords. 

Observe that the web is not a data base, merely a data bank, since the data on 
the web are not structured as data of a data base. 

Citing [15], the idea of the method is as follows: 

When the Google search engine is used to search for the word x, 
Google dsiplays the number of hits that word x has. The ratio of 
this number to the total number of webpages indexed by Google 
represents the probability that word x appears on a webpage [...] 
If word y has a higher conditional probability to appear on a web 
page, given that word x also appears on the webpage, than it does 
by itself, then it can be concluded that words x and y are related. 

Let's cite an example from Cilibrasi and Vitany [lOj which we complete and up- 
date the figures. The searches for the index term "horse" , "rider" and "molecule" 
respectively return 156, 62.2 and 45.6 million hits. Searches for pairs of words 
"horse rider" and "horse molecule" respectively return 2.66 and 1.52 million 
hits. These figures stress a stronger relation between the words "horse" and 
"rider" than between "horse" and "molecule" . 

Another example with famous paintings: "Dejeuner sur I'herbe" , "Moulin de la 
Galette" and "la Joconde" . Let refer them by a, b, c. Google searches for a, b, 
c respectively give 446 000, 278 000 and 1 310 000 hits. As both the searches 
for a+b, a+c and b+c, they respectively give 13 700, 888 and 603 hits. Clearly, 
the two paintings by Renoir are more often cited together than each one is with 
the painting by da Vinci. 

In this way, the method regroups paintings by artists, using what is said about 
these paintings on the web. But this does not associate the painters to groups 
of paintings. 

Formally, Cilibrasi and Vitany [lOl [11] define the normalized Google distance 
as follows: 

Arr^nf ^ max(log /(x), log /(y)) - log /(x, y) 

NGD[x,y) = — — — — — — — — 

logM - mm(log/(2;),log/(y)) 

where /(zi, ...) is the number of hits for the conjunctive query zi, ... and M is 
the total number of webpages that Google indexes. 
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5.3.2 Discussing the method 

1. The number of objects in a future classification and that of canonical repre- 
sentatives of the different corpus is not chosen in advance nor even boundable in 
advance and it is constantly moving. This dynamical and uncontrolled feature 
is a totally new experience. 

2. Domains a priori completely rebel to classification as is the pictorial domain 
(no normalization of paintings is possible) can now be considered. Because we 
are no more dealing with the paintings themselves but with what is said about 
them on the web. And, whereas the "pictorial language" is merely a metaphor, 
this is a true "language" which deals with keywords and their relations in the 
texts written by web users. 

3. However, there is a big limitation to the method, that of a closed world: the 

World according to Google, Information according to Google... 

If Google finds something, one can check its pertinence. Else, what does it 

mean? Sole certainty, that of uncertainty. 

When failing to get hits with several keywords, we give up the original query 
and modify it up to the point Google gives some pertinent answers. 
So that failure is as negation in Prolog which is much weaker than classical 
negation. It's reasonable to give up a query and accordingly consider the re- 
lated conjunction as meaningless. However, one should keep in mind that this 
is relative to the close - and relatively small - world of data on the web, the sole 
world accessible to Google. 

When succeeding with a query, the risk is to stop on this succeeding query and 

- forget that previous queries have been tried which failed, 

- omit going on with some other queries which could possibly lead to more per- 
tinent answers. 

There is a need to formalize information on the web and the relations ruling the 
data it contains. And also the notion of pertinence. A mathematical framework 

is badly needed. 

This remarkable innovative approach is still in its infancy. 
5.4 Some final remEirks 

These approaches to classification via compression and Google search of the web 
are really provocative. They allow for classification of diverse corpus along a 
top-down operational mode as opposed to bottom-up grouping. 
Top-down since there is no prerequisite of any a priori knowledge of the content 
of the texts under consideration. One gets information on the texts without 
entering their semantics, simply by compressing them or counting hits with 
Google. This has much resemblance with statistical methods which point cor- 
relations to group objects. Indeed, compressors and Google use a large amount 
of statistical expertise. 

On the opposite, a botton-up approach uses keywords which have to be previ- 
ously known so that we already have in mind what the groups of the classification 
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should be. 



Let's illustrate this top-down versus bottom-up opposition by contrasting three 
approaehes related to the classical comprehension schema. 

Mathematical approach. 

This is a global, intrinsically deterministic approach along a fundamental di- 
chotomy: true/false, provable/inconsistent. A quest for absoluteness based on 
certainty. This is reflected in the classical comprehension schema 

Vy3Z Z = {x€y\V{x)} 

where P is a property fixed in advance. 

Probabilistic approach. 

In this pragmatic approach uncertainty is taken into consideration, it is bounded 
and treated mathematically. This can be related to a probabilistic version of the 
comprehension schema where the truth of Vix) is replaced by some limitation of 
the uncertainty: the probability that x satisfies V is true is in a given interval. 
Which asks for a two arguments property V : 

Vy 3Z Z = {xey\ ii{{uj G O | Pix.uj)}) e /} 

where /u is a probability on some space 17 and I is some interval of [0, 1]. 

The above mathematical and probabilistic approaches are bottom-up. One 
starts with a given V to group objects. 

Google approach. 

Now, there is no idea of the interval of uncertainty. Google may give 0% up to 
100% of pertinent answers. It seems to be much harder to put in a mathematical 
framework. But this is quite an exciting approach, one of the few top-down ones 
together with the compression approach and those based on statistical inference. 
This Google approach reveals properties, regularity laws. 
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